The SDQZ-Pipe Technique: Doing More With Less, Faster and Cheaper Reading UNIX-Compressed SAS Data Sets Directly in a Data Step

While hard disk space is amazingly "cheap" nowadays, many of us programmers continue to find ourselves working in environments that have significant hard disk constraints. The ’always-ever-increasing-in-size’ data set too quickly eats up the hard disk "real estate" available to us which will give us immediate and direct access to our data. How many of us have found ourselves spending an inordinate amount of time on trivial, mundane, and unwelcomed tasks such as temporarily compressing or moving our data offline, slicing-and-dicing it, or making some other sort of concession with respect to our data just because there was not enough hard disk space for what we needed to do? As a result, in order to save disk space, programmers working in a UNIX environment will typically store SAS data sets in UNIX-compressed format. Unfortunately, SAS is unable to directly read a UNIX-compressed, random access, SAS data set. The typical “INFILE...PIPE” approach does not work with SAS random access (ssd01) files. Programmers are instead forced to uncompress SAS data sets in an entirely separate step prior to reading the data in SAS. The technique presented in this paper however reduces the processing of a UNIX-compressed SAS data set to one step using named pipes and the TAPE sequential engine and, for this reason, it should prove to be a valuable tool for any programmer’s toolbox.